Introduction

The main objective of this notebook is to perform several different matching methods in experiment 3 (which was the experiment that presented the best results in the previous work) to select the best model.

Model Selection

In short, the prior results showed that the model using Nearest Neighbor matching with stratified sampling outperformed the other models. In particular, the experiment 3 (regarding to the results of feature selection with stratified sampling, considering the top 5 features) presented the best results for most of the tested models. However, the results could be further improved. So, before move on to the last phase (the model validation), we can try using other matching strategies. Now that we have selected the best expirement.

Match using Coarsened Exact Matching (CEM): Small Dataset

## 
## Using 'treat'='1' as baseline group
## 
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x_sm, 
##     method = "cem")
## 
## Summary of balance for all data:
##                                   Means Treated Means Control SD Control
## distance                                 0.6201        0.3793     0.2011
## daily_num_sessions_started               2.8624        2.3903     2.7452
## daily_num_sessions_started_max           5.1994        4.3137     4.8323
## FX_PAGE_LOAD_MS_2_PARENT              3044.1381     3459.4753  1918.2850
## memory_mb                             9375.4970     8954.2914  7959.4119
## num_active_days                          5.5615        5.3423     2.2659
## num_addons                               5.6684        7.8676     3.3668
## num_bookmarks                          164.7282      244.0887  1285.4479
## profile_age                            899.2399      893.4752   761.9573
## session_length                           9.2079       12.2015    14.4045
## session_length_max                      18.1451       22.5676    29.8243
## TIME_TO_DOM_COMPLETE_MS               3304.4993     4397.6110  4283.8936
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2291.1852     2743.6285  2737.7315
## TIME_TO_DOM_INTERACTIVE_MS            1797.9580     2411.6625  2431.1910
## TIME_TO_LOAD_EVENT_END_MS             3026.7281     4134.4538  4026.6727
## TIME_TO_NON_BLANK_PAINT_MS            1457.4383     1837.6274  2148.1641
##                                    Mean Diff  eQQ Med  eQQ Mean     eQQ Max
## distance                              0.2408   0.2500    0.2408      0.3106
## daily_num_sessions_started            0.4721   0.4000    0.4737      1.5750
## daily_num_sessions_started_max        0.8857   1.0000    0.8882      3.0000
## FX_PAGE_LOAD_MS_2_PARENT           -415.3372 286.3527  415.2519   1217.0293
## memory_mb                           421.2056  31.0000  542.1348 196547.0000
## num_active_days                       0.2192   0.0000    0.2374      1.0000
## num_addons                           -2.1992   2.0000    2.1994    126.6667
## num_bookmarks                       -79.3605   1.0000   79.1581  21769.0000
## profile_age                           5.7647  23.0000   25.2873   1369.0000
## session_length                       -2.9936   1.4564    2.9903    151.2126
## session_length_max                   -4.4224   2.8661    4.4036    858.3503
## TIME_TO_DOM_COMPLETE_MS           -1093.1117 408.7792 1092.1439  11538.8571
## TIME_TO_DOM_CONTENT_LOADED_END_MS  -452.4433 218.3252  451.4889  12439.4915
## TIME_TO_DOM_INTERACTIVE_MS         -613.7044 236.0083  612.6128  20607.0000
## TIME_TO_LOAD_EVENT_END_MS         -1107.7257 428.3251 1106.7063  11196.2118
## TIME_TO_NON_BLANK_PAINT_MS         -380.1891 151.5375  379.0529  16102.5273
## 
## 
## Summary of balance for matched data:
##                                   Means Treated Means Control SD Control
## distance                                 0.6378        0.4615     0.1557
## daily_num_sessions_started               2.2732        2.2302     1.8198
## daily_num_sessions_started_max           4.1538        4.0978     3.3454
## FX_PAGE_LOAD_MS_2_PARENT              2575.2467     2574.2537  1117.0086
## memory_mb                             9345.2703     8931.3493  5934.1611
## num_active_days                          5.6123        5.6123     2.0231
## num_addons                               5.4839        7.0581     1.6139
## num_bookmarks                          102.9048      107.0281   266.4720
## profile_age                            776.3038      767.3874   682.0415
## session_length                           8.9416        9.0935     8.7593
## session_length_max                      17.2511       17.3970    16.8420
## TIME_TO_DOM_COMPLETE_MS               2541.2212     2571.6745  1418.3503
## TIME_TO_DOM_CONTENT_LOADED_END_MS     1645.3293     1643.8191   920.1184
## TIME_TO_DOM_INTERACTIVE_MS            1434.9835     1456.7013   775.5078
## TIME_TO_LOAD_EVENT_END_MS             2390.2772     2439.3777  1342.7873
## TIME_TO_NON_BLANK_PAINT_MS            1153.8016     1167.0823   650.7752
##                                   Mean Diff  eQQ Med eQQ Mean    eQQ Max
## distance                             0.1763   0.2110   0.2028     0.2391
## daily_num_sessions_started           0.0430   0.2857   0.2879     2.9583
## daily_num_sessions_started_max       0.0559   0.0000   0.4787     3.0000
## FX_PAGE_LOAD_MS_2_PARENT             0.9930  84.1227 104.1835   676.1569
## memory_mb                          413.9210  18.0000 350.8810 16219.0000
## num_active_days                      0.0000   0.0000   0.1566     1.0000
## num_addons                          -1.5742   2.0000   1.8460     4.0000
## num_bookmarks                       -4.1233   2.0000  15.3790   334.6000
## profile_age                          8.9163  47.0000  50.4190   252.0000
## session_length                      -0.1519   0.8009   1.4499    10.8062
## session_length_max                  -0.1459   1.8244   1.9894    10.1356
## TIME_TO_DOM_COMPLETE_MS            -30.4533 138.3078 187.2241  1005.1325
## TIME_TO_DOM_CONTENT_LOADED_END_MS    1.5102  70.1490  88.3670  1294.2653
## TIME_TO_DOM_INTERACTIVE_MS         -21.7178  82.8923 110.1197   870.8117
## TIME_TO_LOAD_EVENT_END_MS          -49.1004 156.5560 202.9708  1173.3746
## TIME_TO_NON_BLANK_PAINT_MS         -13.2806  54.5940  76.3997  1754.5214
## 
## Percent Balance Improvement:
##                                   Mean Diff.   eQQ Med eQQ Mean  eQQ Max
## distance                             26.8023   15.6105  15.7702  23.0370
## daily_num_sessions_started           90.9009   28.5714  39.2304 -87.8307
## daily_num_sessions_started_max       93.6847  100.0000  46.1049   0.0000
## FX_PAGE_LOAD_MS_2_PARENT             99.7609   70.6227  74.9108  44.4420
## memory_mb                             1.7295   41.9355  35.2779  91.7480
## num_active_days                     100.0000    0.0000  34.0033   0.0000
## num_addons                           28.4206    0.0000  16.0687  96.8421
## num_bookmarks                        94.8043 -100.0000  80.5718  98.4630
## profile_age                         -54.6712 -104.3478 -99.3845  81.5924
## session_length                       94.9249   45.0069  51.5132  92.8536
## session_length_max                   96.7003   36.3442  54.8230  98.8192
## TIME_TO_DOM_COMPLETE_MS              97.2141   66.1656  82.8572  91.2891
## TIME_TO_DOM_CONTENT_LOADED_END_MS    99.6662   67.8695  80.4276  89.5955
## TIME_TO_DOM_INTERACTIVE_MS           96.4612   64.8774  82.0246  95.7742
## TIME_TO_LOAD_EVENT_END_MS            95.5675   63.4493  81.6599  89.5199
## TIME_TO_NON_BLANK_PAINT_MS           96.5068   63.9732  79.8446  89.1041
## 
## Sample sizes:
##           Control Treated
## All         35027   34973
## Matched     19560   23021
## Unmatched   15467   11952
## Discarded       0       0

Match using Coarsened Exact Matching (CEM): Full Dataset

## 
## Using 'treat'='1' as baseline group
## 
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x, 
##     method = "cem")
## 
## Summary of balance for all data:
##                                   Means Treated Means Control SD Control
## distance                                 0.6213        0.3787     0.2009
## daily_num_sessions_started               2.8876        2.3689     2.7099
## daily_num_sessions_started_max           5.2501        4.2814     4.8046
## FX_PAGE_LOAD_MS_2_PARENT              3027.5228     3463.7084  1920.1256
## memory_mb                             9460.0891     8965.1561  7925.6741
## num_active_days                          5.5746        5.3462     2.2644
## num_addons                               5.6632        7.8554     3.3349
## num_bookmarks                          164.1029      242.4878  1292.6145
## profile_age                            893.9339      893.7534   762.4791
## session_length                           9.2495       12.2962    14.7435
## session_length_max                      18.1985       22.7066    30.2272
## TIME_TO_DOM_COMPLETE_MS               3285.4718     4388.9143  4254.1363
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2292.5533     2737.6381  2700.5353
## TIME_TO_DOM_INTERACTIVE_MS            1792.4896     2404.3932  2397.0177
## TIME_TO_LOAD_EVENT_END_MS             3012.8713     4126.9123  4007.0480
## TIME_TO_NON_BLANK_PAINT_MS            1449.9423     1833.6557  2128.4506
##                                    Mean Diff  eQQ Med  eQQ Mean     eQQ Max
## distance                              0.2426   0.2527    0.2426      0.3114
## daily_num_sessions_started            0.5188   0.4167    0.5190      1.7500
## daily_num_sessions_started_max        0.9687   1.0000    0.9692     11.0000
## FX_PAGE_LOAD_MS_2_PARENT           -436.1856 299.6662  436.2548   1250.9746
## memory_mb                           494.9330  31.0000  555.3761 262176.0000
## num_active_days                       0.2284   0.0000    0.2413      1.0000
## num_addons                           -2.1922   2.0000    2.1941    109.0000
## num_bookmarks                       -78.3849   1.0000   79.0181  21769.0000
## profile_age                           0.1805  22.0000   23.1161   1366.0000
## session_length                       -3.0467   1.4538    3.0489    151.4597
## session_length_max                   -4.5080   2.7333    4.5187    871.0939
## TIME_TO_DOM_COMPLETE_MS           -1103.4424 421.7080 1103.4946  12182.3743
## TIME_TO_DOM_CONTENT_LOADED_END_MS  -445.0848 229.0178  445.1832  10634.8539
## TIME_TO_DOM_INTERACTIVE_MS         -611.9036 243.7872  612.0582  19062.0000
## TIME_TO_LOAD_EVENT_END_MS         -1114.0410 441.3178 1114.0691  11196.2118
## TIME_TO_NON_BLANK_PAINT_MS         -383.7134 155.3258  384.1746  19260.1667
## 
## 
## Summary of balance for matched data:
##                                   Means Treated Means Control SD Control
## distance                                 0.6368        0.4606     0.1581
## daily_num_sessions_started               2.3597        2.3181     1.9440
## daily_num_sessions_started_max           4.3248        4.2638     3.5848
## FX_PAGE_LOAD_MS_2_PARENT              2630.8495     2629.8182  1165.2841
## memory_mb                             9345.9336     9007.7607  6130.1538
## num_active_days                          5.6190        5.6190     2.0238
## num_addons                               5.4908        7.0621     1.6316
## num_bookmarks                          106.6405      108.9676   285.6003
## profile_age                            796.3677      786.3890   696.6420
## session_length                           9.0017        9.1066     8.9578
## session_length_max                      17.4904       17.4516    17.3295
## TIME_TO_DOM_COMPLETE_MS               2610.7773     2636.8150  1491.7543
## TIME_TO_DOM_CONTENT_LOADED_END_MS     1697.3437     1693.6141   997.9611
## TIME_TO_DOM_INTERACTIVE_MS            1469.5869     1489.4234   820.8763
## TIME_TO_LOAD_EVENT_END_MS             2448.9974     2494.9380  1408.4943
## TIME_TO_NON_BLANK_PAINT_MS            1180.7099     1191.0518   681.9404
##                                   Mean Diff  eQQ Med eQQ Mean    eQQ Max
## distance                             0.1762   0.2175   0.2092     0.2479
## daily_num_sessions_started           0.0416   0.3333   0.3373     1.3750
## daily_num_sessions_started_max       0.0610   1.0000   0.5791     3.0000
## FX_PAGE_LOAD_MS_2_PARENT             1.0313  88.2880 115.8262  1222.0000
## memory_mb                          338.1729  14.0000 309.7014 16281.0000
## num_active_days                      0.0000   0.0000   0.1516     1.0000
## num_addons                          -1.5713   2.0000   1.9054     4.8333
## num_bookmarks                       -2.3271   2.0000  17.9224  2354.8750
## profile_age                          9.9787  36.0000  43.8503   246.0000
## session_length                      -0.1049   0.9890   1.5868     6.6953
## session_length_max                   0.0388   1.9725   2.1093    30.2350
## TIME_TO_DOM_COMPLETE_MS            -26.0378 139.8803 200.9518  1109.0728
## TIME_TO_DOM_CONTENT_LOADED_END_MS    3.7296  73.2474  96.4370  1088.3989
## TIME_TO_DOM_INTERACTIVE_MS         -19.8365  85.7666 120.1839  1005.4136
## TIME_TO_LOAD_EVENT_END_MS          -45.9406 159.2815 221.6872  3788.0222
## TIME_TO_NON_BLANK_PAINT_MS         -10.3419  55.7787  79.7257  1161.2917
## 
## Percent Balance Improvement:
##                                   Mean Diff.   eQQ Med eQQ Mean eQQ Max
## distance                             27.3811   13.9260  13.7551 20.3929
## daily_num_sessions_started           91.9833   20.0000  35.0104 21.4286
## daily_num_sessions_started_max       93.7068    0.0000  40.2527 72.7273
## FX_PAGE_LOAD_MS_2_PARENT             99.7636   70.5379  73.4499  2.3162
## memory_mb                            31.6730   54.8387  44.2357 93.7900
## num_active_days                     100.0000    0.0000  37.1827  0.0000
## num_addons                           28.3232    0.0000  13.1586 95.5657
## num_bookmarks                        97.0312 -100.0000  77.3186 89.1824
## profile_age                       -5427.1469  -63.6364 -89.6960 81.9912
## session_length                       96.5556   31.9662  47.9543 95.5795
## session_length_max                   99.1398   27.8354  53.3211 96.5291
## TIME_TO_DOM_COMPLETE_MS              97.6403   66.8301  81.7895 90.8961
## TIME_TO_DOM_CONTENT_LOADED_END_MS    99.1620   68.0167  78.3377 89.7657
## TIME_TO_DOM_INTERACTIVE_MS           96.7582   64.8191  80.3640 94.7256
## TIME_TO_LOAD_EVENT_END_MS            95.8762   63.9078  80.1011 66.1669
## TIME_TO_NON_BLANK_PAINT_MS           97.3048   64.0892  79.2475 93.9705
## 
## Sample sizes:
##           Control Treated
## All         59627   59627
## Matched     36765   42779
## Unmatched   22862   16848
## Discarded       0       0

Match using Nearest Neighbor matching: Small Dataset

## 
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x_sm, 
##     method = "nearest", replace = TRUE)
## 
## Summary of balance for all data:
##                                   Means Treated Means Control SD Control
## distance                                 0.6201        0.3793     0.2011
## daily_num_sessions_started               2.8624        2.3903     2.7452
## daily_num_sessions_started_max           5.1994        4.3137     4.8323
## FX_PAGE_LOAD_MS_2_PARENT              3044.1381     3459.4753  1918.2850
## memory_mb                             9375.4970     8954.2914  7959.4119
## num_active_days                          5.5615        5.3423     2.2659
## num_addons                               5.6684        7.8676     3.3668
## num_bookmarks                          164.7282      244.0887  1285.4479
## profile_age                            899.2399      893.4752   761.9573
## session_length                           9.2079       12.2015    14.4045
## session_length_max                      18.1451       22.5676    29.8243
## TIME_TO_DOM_COMPLETE_MS               3304.4993     4397.6110  4283.8936
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2291.1852     2743.6285  2737.7315
## TIME_TO_DOM_INTERACTIVE_MS            1797.9580     2411.6625  2431.1910
## TIME_TO_LOAD_EVENT_END_MS             3026.7281     4134.4538  4026.6727
## TIME_TO_NON_BLANK_PAINT_MS            1457.4383     1837.6274  2148.1641
##                                    Mean Diff  eQQ Med  eQQ Mean     eQQ Max
## distance                              0.2408   0.2500    0.2408      0.3106
## daily_num_sessions_started            0.4721   0.4000    0.4737      1.5750
## daily_num_sessions_started_max        0.8857   1.0000    0.8882      3.0000
## FX_PAGE_LOAD_MS_2_PARENT           -415.3372 286.3527  415.2519   1217.0293
## memory_mb                           421.2056  31.0000  542.1348 196547.0000
## num_active_days                       0.2192   0.0000    0.2374      1.0000
## num_addons                           -2.1992   2.0000    2.1994    126.6667
## num_bookmarks                       -79.3605   1.0000   79.1581  21769.0000
## profile_age                           5.7647  23.0000   25.2873   1369.0000
## session_length                       -2.9936   1.4564    2.9903    151.2126
## session_length_max                   -4.4224   2.8661    4.4036    858.3503
## TIME_TO_DOM_COMPLETE_MS           -1093.1117 408.7792 1092.1439  11538.8571
## TIME_TO_DOM_CONTENT_LOADED_END_MS  -452.4433 218.3252  451.4889  12439.4915
## TIME_TO_DOM_INTERACTIVE_MS         -613.7044 236.0083  612.6128  20607.0000
## TIME_TO_LOAD_EVENT_END_MS         -1107.7257 428.3251 1106.7063  11196.2118
## TIME_TO_NON_BLANK_PAINT_MS         -380.1891 151.5375  379.0529  16102.5273
## 
## 
## Summary of balance for matched data:
##                                   Means Treated Means Control SD Control
## distance                                 0.6201        0.6201     0.1995
## daily_num_sessions_started               2.8624        4.0732     4.9384
## daily_num_sessions_started_max           5.1994        7.6838     9.8565
## FX_PAGE_LOAD_MS_2_PARENT              3044.1381     2912.7561  1622.9078
## memory_mb                             9375.4970    14598.4850 18130.7604
## num_active_days                          5.5615        5.9370     2.0637
## num_addons                               5.6684        6.2446     1.9905
## num_bookmarks                          164.7282      851.4996  3905.5180
## profile_age                            899.2399      949.5171   809.2864
## session_length                           9.2079        8.1689    10.8524
## session_length_max                      18.1451       17.1514    39.7136
## TIME_TO_DOM_COMPLETE_MS               3304.4993     3195.2355  3352.6550
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2291.1852     2805.0239  3838.0612
## TIME_TO_DOM_INTERACTIVE_MS            1797.9580     1726.9272  1763.8950
## TIME_TO_LOAD_EVENT_END_MS             3026.7281     2854.4041  2673.1022
## TIME_TO_NON_BLANK_PAINT_MS            1457.4383     1397.7523  1622.3912
##                                    Mean Diff  eQQ Med eQQ Mean     eQQ Max
## distance                              0.0000   0.1645   0.1589      0.2015
## daily_num_sessions_started           -1.2108   0.2321   0.2508      2.8750
## daily_num_sessions_started_max       -2.4844   0.0000   0.4174      9.0000
## FX_PAGE_LOAD_MS_2_PARENT            131.3820 119.0923 208.8125    912.0536
## memory_mb                         -5222.9880  29.0000 727.7257 196581.0000
## num_active_days                      -0.3755   0.0000   0.1748      1.0000
## num_addons                           -0.5762   1.5000   1.4821     11.0000
## num_bookmarks                      -686.7714   2.0000  81.0448  21802.0000
## profile_age                         -50.2772  13.0000  19.4053   1597.0000
## session_length                        1.0390   0.2017   1.4697    144.9158
## session_length_max                    0.9938   0.6333   2.3059    858.3503
## TIME_TO_DOM_COMPLETE_MS             109.2638 147.9601 562.4198  11171.1250
## TIME_TO_DOM_CONTENT_LOADED_END_MS  -513.8387  88.5859 298.0847  12439.4915
## TIME_TO_DOM_INTERACTIVE_MS           71.0308  78.5130 318.7461  20607.0000
## TIME_TO_LOAD_EVENT_END_MS           172.3240 147.1974 563.7810   7937.8281
## TIME_TO_NON_BLANK_PAINT_MS           59.6860  56.2857 202.8511   9919.0000
## 
## Percent Balance Improvement:
##                                   Mean Diff.   eQQ Med eQQ Mean   eQQ Max
## distance                             99.9987   34.2278  34.0267   35.1400
## daily_num_sessions_started         -156.4632   41.9643  47.0445  -82.5397
## daily_num_sessions_started_max     -180.5042  100.0000  53.0022 -200.0000
## FX_PAGE_LOAD_MS_2_PARENT             68.3674   58.4106  49.7142   25.0590
## memory_mb                         -1140.0092    6.4516 -34.2334   -0.0173
## num_active_days                     -71.2910    0.0000  26.3693    0.0000
## num_addons                           73.7992   25.0000  32.6121   91.3158
## num_bookmarks                      -765.3816 -100.0000  -2.3834   -0.1516
## profile_age                        -772.1575   43.4783  23.2609  -16.6545
## session_length                       65.2940   86.1510  50.8513    4.1642
## session_length_max                   77.5289   77.9027  47.6370    0.0000
## TIME_TO_DOM_COMPLETE_MS              90.0043   63.8044  48.5031    3.1869
## TIME_TO_DOM_CONTENT_LOADED_END_MS   -13.5698   59.4248  33.9774    0.0000
## TIME_TO_DOM_INTERACTIVE_MS           88.4259   66.7330  47.9694    0.0000
## TIME_TO_LOAD_EVENT_END_MS            84.4434   65.6342  49.0578   29.1026
## TIME_TO_NON_BLANK_PAINT_MS           84.3010   62.8569  46.4848   38.4010
## 
## Sample sizes:
##           Control Treated
## All         35027   34973
## Matched     12491   34973
## Unmatched   22536       0
## Discarded       0       0

Match using Nearest Neighbor matching: Full Dataset

## 
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x, 
##     method = "nearest", replace = TRUE)
## 
## Summary of balance for all data:
##                                   Means Treated Means Control SD Control
## distance                                 0.6213        0.3787     0.2009
## daily_num_sessions_started               2.8876        2.3689     2.7099
## daily_num_sessions_started_max           5.2501        4.2814     4.8046
## FX_PAGE_LOAD_MS_2_PARENT              3027.5228     3463.7084  1920.1256
## memory_mb                             9460.0891     8965.1561  7925.6741
## num_active_days                          5.5746        5.3462     2.2644
## num_addons                               5.6632        7.8554     3.3349
## num_bookmarks                          164.1029      242.4878  1292.6145
## profile_age                            893.9339      893.7534   762.4791
## session_length                           9.2495       12.2962    14.7435
## session_length_max                      18.1985       22.7066    30.2272
## TIME_TO_DOM_COMPLETE_MS               3285.4718     4388.9143  4254.1363
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2292.5533     2737.6381  2700.5353
## TIME_TO_DOM_INTERACTIVE_MS            1792.4896     2404.3932  2397.0177
## TIME_TO_LOAD_EVENT_END_MS             3012.8713     4126.9123  4007.0480
## TIME_TO_NON_BLANK_PAINT_MS            1449.9423     1833.6557  2128.4506
##                                    Mean Diff  eQQ Med  eQQ Mean     eQQ Max
## distance                              0.2426   0.2527    0.2426      0.3114
## daily_num_sessions_started            0.5188   0.4167    0.5190      1.7500
## daily_num_sessions_started_max        0.9687   1.0000    0.9692     11.0000
## FX_PAGE_LOAD_MS_2_PARENT           -436.1856 299.6662  436.2548   1250.9746
## memory_mb                           494.9330  31.0000  555.3761 262176.0000
## num_active_days                       0.2284   0.0000    0.2413      1.0000
## num_addons                           -2.1922   2.0000    2.1941    109.0000
## num_bookmarks                       -78.3849   1.0000   79.0181  21769.0000
## profile_age                           0.1805  22.0000   23.1161   1366.0000
## session_length                       -3.0467   1.4538    3.0489    151.4597
## session_length_max                   -4.5080   2.7333    4.5187    871.0939
## TIME_TO_DOM_COMPLETE_MS           -1103.4424 421.7080 1103.4946  12182.3743
## TIME_TO_DOM_CONTENT_LOADED_END_MS  -445.0848 229.0178  445.1832  10634.8539
## TIME_TO_DOM_INTERACTIVE_MS         -611.9036 243.7872  612.0582  19062.0000
## TIME_TO_LOAD_EVENT_END_MS         -1114.0410 441.3178 1114.0691  11196.2118
## TIME_TO_NON_BLANK_PAINT_MS         -383.7134 155.3258  384.1746  19260.1667
## 
## 
## Summary of balance for matched data:
##                                   Means Treated Means Control SD Control
## distance                                 0.6213        0.6213     0.2000
## daily_num_sessions_started               2.8876        4.3582     5.2354
## daily_num_sessions_started_max           5.2501        8.0823     9.5927
## FX_PAGE_LOAD_MS_2_PARENT              3027.5228     2894.2370  1635.9350
## memory_mb                             9460.0891    14479.9604 18183.1455
## num_active_days                          5.5746        5.9842     2.0373
## num_addons                               5.6632        6.2566     2.0935
## num_bookmarks                          164.1029      603.6715  3468.5644
## profile_age                            893.9339      921.9102   805.7037
## session_length                           9.2495        8.2741    12.0331
## session_length_max                      18.1985       17.9987    50.5116
## TIME_TO_DOM_COMPLETE_MS               3285.4718     3128.3417  3058.5441
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2292.5533     2820.1932  3950.7895
## TIME_TO_DOM_INTERACTIVE_MS            1792.4896     1712.4133  1782.8317
## TIME_TO_LOAD_EVENT_END_MS             3012.8713     2816.9697  2652.6928
## TIME_TO_NON_BLANK_PAINT_MS            1449.9423     1410.9626  1885.5897
##                                    Mean Diff  eQQ Med eQQ Mean     eQQ Max
## distance                              0.0000   0.1655   0.1594      0.2022
## daily_num_sessions_started           -1.4706   0.2083   0.2345      2.4643
## daily_num_sessions_started_max       -2.8322   0.0000   0.4117     10.0000
## FX_PAGE_LOAD_MS_2_PARENT            133.2858 119.5299 219.7080    977.6778
## memory_mb                         -5019.8713  27.0000 682.6319 262176.0000
## num_active_days                      -0.4096   0.0000   0.1577      1.0000
## num_addons                           -0.5934   1.5000   1.4923    109.0000
## num_bookmarks                      -439.5686   1.6667  79.1675  21761.6250
## profile_age                         -27.9763  21.0000  25.2114   1231.0000
## session_length                        0.9754   0.1928   1.3577    140.8883
## session_length_max                    0.1998   0.5817   2.1986    763.0081
## TIME_TO_DOM_COMPLETE_MS             157.1302 160.5541 573.2693  11538.8571
## TIME_TO_DOM_CONTENT_LOADED_END_MS  -527.6399  92.3230 284.1487  10634.8539
## TIME_TO_DOM_INTERACTIVE_MS           80.0763  88.6948 324.3294  18812.3581
## TIME_TO_LOAD_EVENT_END_MS           195.9016 158.7490 568.8458  11171.1250
## TIME_TO_NON_BLANK_PAINT_MS           38.9797  61.6206 212.4963  17714.0000
## 
## Percent Balance Improvement:
##                                    Mean Diff.  eQQ Med eQQ Mean  eQQ Max
## distance                              99.9995  34.5097  34.2993  35.0577
## daily_num_sessions_started          -183.4842  50.0000  54.8258 -40.8163
## daily_num_sessions_started_max      -192.3720 100.0000  57.5216   9.0909
## FX_PAGE_LOAD_MS_2_PARENT              69.4429  60.1123  49.6377  21.8467
## memory_mb                           -914.2527  12.9032 -22.9134   0.0000
## num_active_days                      -79.3392   0.0000  34.6539   0.0000
## num_addons                            72.9305  25.0000  31.9864   0.0000
## num_bookmarks                       -460.7820 -66.6667  -0.1891   0.0339
## profile_age                       -15395.9870   4.5455  -9.0644   9.8829
## session_length                        67.9842  86.7345  55.4693   6.9797
## session_length_max                    95.5681  78.7196  51.3448  12.4081
## TIME_TO_DOM_COMPLETE_MS               85.7600  61.9277  48.0496   5.2824
## TIME_TO_DOM_CONTENT_LOADED_END_MS    -18.5482  59.6874  36.1726   0.0000
## TIME_TO_DOM_INTERACTIVE_MS            86.9136  63.6180  47.0100   1.3096
## TIME_TO_LOAD_EVENT_END_MS             82.4152  64.0284  48.9398   0.2241
## TIME_TO_NON_BLANK_PAINT_MS            89.8414  60.3282  44.6876   8.0278
## 
## Sample sizes:
##           Control Treated
## All         59627   59627
## Matched     21147   59627
## Unmatched   38480       0
## Discarded       0       0

Match using Nearest Neighbor matching: Small Dataset - Mahalanobis

## 
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x_sm, 
##     method = "nearest", distance = "mahalanobis")
## 
## Summary of balance for all data:
##                                   Means Treated Means Control SD Control
## daily_num_sessions_started               2.8624        2.3903     2.7452
## daily_num_sessions_started_max           5.1994        4.3137     4.8323
## FX_PAGE_LOAD_MS_2_PARENT              3044.1381     3459.4753  1918.2850
## memory_mb                             9375.4970     8954.2914  7959.4119
## num_active_days                          5.5615        5.3423     2.2659
## num_addons                               5.6684        7.8676     3.3668
## num_bookmarks                          164.7282      244.0887  1285.4479
## profile_age                            899.2399      893.4752   761.9573
## session_length                           9.2079       12.2015    14.4045
## session_length_max                      18.1451       22.5676    29.8243
## TIME_TO_DOM_COMPLETE_MS               3304.4993     4397.6110  4283.8936
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2291.1852     2743.6285  2737.7315
## TIME_TO_DOM_INTERACTIVE_MS            1797.9580     2411.6625  2431.1910
## TIME_TO_LOAD_EVENT_END_MS             3026.7281     4134.4538  4026.6727
## TIME_TO_NON_BLANK_PAINT_MS            1457.4383     1837.6274  2148.1641
##                                    Mean Diff  eQQ Med  eQQ Mean     eQQ Max
## daily_num_sessions_started            0.4721   0.4000    0.4737      1.5750
## daily_num_sessions_started_max        0.8857   1.0000    0.8882      3.0000
## FX_PAGE_LOAD_MS_2_PARENT           -415.3372 286.3527  415.2519   1217.0293
## memory_mb                           421.2056  31.0000  542.1348 196547.0000
## num_active_days                       0.2192   0.0000    0.2374      1.0000
## num_addons                           -2.1992   2.0000    2.1994    126.6667
## num_bookmarks                       -79.3605   1.0000   79.1581  21769.0000
## profile_age                           5.7647  23.0000   25.2873   1369.0000
## session_length                       -2.9936   1.4564    2.9903    151.2126
## session_length_max                   -4.4224   2.8661    4.4036    858.3503
## TIME_TO_DOM_COMPLETE_MS           -1093.1117 408.7792 1092.1439  11538.8571
## TIME_TO_DOM_CONTENT_LOADED_END_MS  -452.4433 218.3252  451.4889  12439.4915
## TIME_TO_DOM_INTERACTIVE_MS         -613.7044 236.0083  612.6128  20607.0000
## TIME_TO_LOAD_EVENT_END_MS         -1107.7257 428.3251 1106.7063  11196.2118
## TIME_TO_NON_BLANK_PAINT_MS         -380.1891 151.5375  379.0529  16102.5273
## 
## 
## Summary of balance for matched data:
##                                   Means Treated Means Control SD Control
## daily_num_sessions_started               2.8624        2.3903     2.7451
## daily_num_sessions_started_max           5.1994        4.3144     4.8332
## FX_PAGE_LOAD_MS_2_PARENT              3044.1381     3456.7463  1914.4418
## memory_mb                             9375.4970     8950.7645  7959.4384
## num_active_days                          5.5615        5.3435     2.2649
## num_addons                               5.6684        7.8592     3.2436
## num_bookmarks                          164.7282      221.2422   987.5952
## profile_age                            899.2399      893.6282   762.0575
## session_length                           9.2079       12.1616    14.1624
## session_length_max                      18.1451       22.3865    26.9638
## TIME_TO_DOM_COMPLETE_MS               3304.4993     4380.9954  4213.0576
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2291.1852     2732.9433  2676.6454
## TIME_TO_DOM_INTERACTIVE_MS            1797.9580     2400.7730  2358.5067
## TIME_TO_LOAD_EVENT_END_MS             3026.7281     4121.5844  3975.2591
## TIME_TO_NON_BLANK_PAINT_MS            1457.4383     1824.7957  1989.4600
##                                    Mean Diff  eQQ Med  eQQ Mean     eQQ Max
## daily_num_sessions_started            0.4721   0.4000    0.4734      1.6250
## daily_num_sessions_started_max        0.8850   1.0000    0.8868      4.0000
## FX_PAGE_LOAD_MS_2_PARENT           -412.6082 285.7252  412.6591   1206.4978
## memory_mb                           424.7325  31.0000  541.6251 196547.0000
## num_active_days                       0.2180   0.0000    0.2365      1.0000
## num_addons                           -2.1908   2.0000    2.1925     26.6667
## num_bookmarks                       -56.5140   1.0000   57.3093   4825.5000
## profile_age                           5.6118  23.0000   25.2124   1385.0000
## session_length                       -2.9538   1.4542    2.9556    151.2126
## session_length_max                   -4.2413   2.8572    4.2510    174.1819
## TIME_TO_DOM_COMPLETE_MS           -1076.4961 408.0023 1076.5944   8525.6667
## TIME_TO_DOM_CONTENT_LOADED_END_MS  -441.7581 217.8830  441.8767   5888.6348
## TIME_TO_DOM_INTERACTIVE_MS         -602.8150 235.7796  602.8969   6066.2727
## TIME_TO_LOAD_EVENT_END_MS         -1094.8563 427.5796 1094.8774   8904.8009
## TIME_TO_NON_BLANK_PAINT_MS         -367.3575 151.2618  370.2076  26903.0000
## 
## Percent Balance Improvement:
##                                   Mean Diff. eQQ Med eQQ Mean  eQQ Max
## daily_num_sessions_started           -0.0029  0.0000   0.0597  -3.1746
## daily_num_sessions_started_max        0.0805  0.0000   0.1545 -33.3333
## FX_PAGE_LOAD_MS_2_PARENT              0.6571  0.2191   0.6244   0.8653
## memory_mb                            -0.8373  0.0000   0.0940   0.0000
## num_active_days                       0.5411  0.0000   0.3614   0.0000
## num_addons                            0.3846  0.0000   0.3148  78.9474
## num_bookmarks                        28.7882  0.0000  27.6015  77.8332
## profile_age                           2.6530  0.0000   0.2964  -1.1687
## session_length                        1.3320  0.1537   1.1626   0.0000
## session_length_max                    4.0949  0.3101   3.4646  79.7074
## TIME_TO_DOM_COMPLETE_MS               1.5200  0.1900   1.4238  26.1134
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2.3617  0.2025   2.1290  52.6618
## TIME_TO_DOM_INTERACTIVE_MS            1.7744  0.0969   1.5860  70.5621
## TIME_TO_LOAD_EVENT_END_MS             1.1618  0.1740   1.0688  20.4659
## TIME_TO_NON_BLANK_PAINT_MS            3.3751  0.1819   2.3335 -67.0732
## 
## Sample sizes:
##           Control Treated
## All         35027   34973
## Matched     34973   34973
## Unmatched      54       0
## Discarded       0       0

Match using Nearest Neighbor matching: Full Dataset - Mahalanobis

## 
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x, 
##     method = "nearest", distance = "mahalanobis")
## 
## Summary of balance for all data:
##                                   Means Treated Means Control SD Control
## daily_num_sessions_started               2.8876        2.3689     2.7099
## daily_num_sessions_started_max           5.2501        4.2814     4.8046
## FX_PAGE_LOAD_MS_2_PARENT              3027.5228     3463.7084  1920.1256
## memory_mb                             9460.0891     8965.1561  7925.6741
## num_active_days                          5.5746        5.3462     2.2644
## num_addons                               5.6632        7.8554     3.3349
## num_bookmarks                          164.1029      242.4878  1292.6145
## profile_age                            893.9339      893.7534   762.4791
## session_length                           9.2495       12.2962    14.7435
## session_length_max                      18.1985       22.7066    30.2272
## TIME_TO_DOM_COMPLETE_MS               3285.4718     4388.9143  4254.1363
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2292.5533     2737.6381  2700.5353
## TIME_TO_DOM_INTERACTIVE_MS            1792.4896     2404.3932  2397.0177
## TIME_TO_LOAD_EVENT_END_MS             3012.8713     4126.9123  4007.0480
## TIME_TO_NON_BLANK_PAINT_MS            1449.9423     1833.6557  2128.4506
##                                    Mean Diff  eQQ Med  eQQ Mean     eQQ Max
## daily_num_sessions_started            0.5188   0.4167    0.5190      1.7500
## daily_num_sessions_started_max        0.9687   1.0000    0.9692     11.0000
## FX_PAGE_LOAD_MS_2_PARENT           -436.1856 299.6662  436.2548   1250.9746
## memory_mb                           494.9330  31.0000  555.3761 262176.0000
## num_active_days                       0.2284   0.0000    0.2413      1.0000
## num_addons                           -2.1922   2.0000    2.1941    109.0000
## num_bookmarks                       -78.3849   1.0000   79.0181  21769.0000
## profile_age                           0.1805  22.0000   23.1161   1366.0000
## session_length                       -3.0467   1.4538    3.0489    151.4597
## session_length_max                   -4.5080   2.7333    4.5187    871.0939
## TIME_TO_DOM_COMPLETE_MS           -1103.4424 421.7080 1103.4946  12182.3743
## TIME_TO_DOM_CONTENT_LOADED_END_MS  -445.0848 229.0178  445.1832  10634.8539
## TIME_TO_DOM_INTERACTIVE_MS         -611.9036 243.7872  612.0582  19062.0000
## TIME_TO_LOAD_EVENT_END_MS         -1114.0410 441.3178 1114.0691  11196.2118
## TIME_TO_NON_BLANK_PAINT_MS         -383.7134 155.3258  384.1746  19260.1667
## 
## 
## Summary of balance for matched data:
##                                   Means Treated Means Control SD Control
## daily_num_sessions_started               2.8876        2.3689     2.7099
## daily_num_sessions_started_max           5.2501        4.2814     4.8046
## FX_PAGE_LOAD_MS_2_PARENT              3027.5228     3463.7084  1920.1256
## memory_mb                             9460.0891     8965.1561  7925.6741
## num_active_days                          5.5746        5.3462     2.2644
## num_addons                               5.6632        7.8554     3.3349
## num_bookmarks                          164.1029      242.4878  1292.6145
## profile_age                            893.9339      893.7534   762.4791
## session_length                           9.2495       12.2962    14.7435
## session_length_max                      18.1985       22.7066    30.2272
## TIME_TO_DOM_COMPLETE_MS               3285.4718     4388.9143  4254.1363
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2292.5533     2737.6381  2700.5353
## TIME_TO_DOM_INTERACTIVE_MS            1792.4896     2404.3932  2397.0177
## TIME_TO_LOAD_EVENT_END_MS             3012.8713     4126.9123  4007.0480
## TIME_TO_NON_BLANK_PAINT_MS            1449.9423     1833.6557  2128.4506
##                                    Mean Diff  eQQ Med  eQQ Mean     eQQ Max
## daily_num_sessions_started            0.5188   0.4167    0.5190      1.7500
## daily_num_sessions_started_max        0.9687   1.0000    0.9692     11.0000
## FX_PAGE_LOAD_MS_2_PARENT           -436.1856 299.6662  436.2548   1250.9746
## memory_mb                           494.9330  31.0000  555.3761 262176.0000
## num_active_days                       0.2284   0.0000    0.2413      1.0000
## num_addons                           -2.1922   2.0000    2.1941    109.0000
## num_bookmarks                       -78.3849   1.0000   79.0181  21769.0000
## profile_age                           0.1805  22.0000   23.1161   1366.0000
## session_length                       -3.0467   1.4538    3.0489    151.4597
## session_length_max                   -4.5080   2.7333    4.5187    871.0939
## TIME_TO_DOM_COMPLETE_MS           -1103.4424 421.7080 1103.4946  12182.3743
## TIME_TO_DOM_CONTENT_LOADED_END_MS  -445.0848 229.0178  445.1832  10634.8539
## TIME_TO_DOM_INTERACTIVE_MS         -611.9036 243.7872  612.0582  19062.0000
## TIME_TO_LOAD_EVENT_END_MS         -1114.0410 441.3178 1114.0691  11196.2118
## TIME_TO_NON_BLANK_PAINT_MS         -383.7134 155.3258  384.1746  19260.1667
## 
## Percent Balance Improvement:
##                                   Mean Diff. eQQ Med eQQ Mean eQQ Max
## daily_num_sessions_started                 0       0        0       0
## daily_num_sessions_started_max             0       0        0       0
## FX_PAGE_LOAD_MS_2_PARENT                   0       0        0       0
## memory_mb                                  0       0        0       0
## num_active_days                            0       0        0       0
## num_addons                                 0       0        0       0
## num_bookmarks                              0       0        0       0
## profile_age                                0       0        0       0
## session_length                             0       0        0       0
## session_length_max                         0       0        0       0
## TIME_TO_DOM_COMPLETE_MS                    0       0        0       0
## TIME_TO_DOM_CONTENT_LOADED_END_MS          0       0        0       0
## TIME_TO_DOM_INTERACTIVE_MS                 0       0        0       0
## TIME_TO_LOAD_EVENT_END_MS                  0       0        0       0
## TIME_TO_NON_BLANK_PAINT_MS                 0       0        0       0
## 
## Sample sizes:
##           Control Treated
## All         59627   59627
## Matched     59627   59627
## Unmatched       0       0
## Discarded       0       0

Match using Subclassification matching: Small Dataset

## 
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x_sm, 
##     method = "subclass", sub.by = "treat")
## Summary of balance for all data:
##                                   Means Treated Means Control  Mean Diff
## distance                                 0.6201        0.3793     0.2408
## daily_num_sessions_started               2.8624        2.3903     0.4721
## daily_num_sessions_started_max           5.1994        4.3137     0.8857
## FX_PAGE_LOAD_MS_2_PARENT              3044.1381     3459.4753  -415.3372
## memory_mb                             9375.4970     8954.2914   421.2056
## num_active_days                          5.5615        5.3423     0.2192
## num_addons                               5.6684        7.8676    -2.1992
## num_bookmarks                          164.7282      244.0887   -79.3605
## profile_age                            899.2399      893.4752     5.7647
## session_length                           9.2079       12.2015    -2.9936
## session_length_max                      18.1451       22.5676    -4.4224
## TIME_TO_DOM_COMPLETE_MS               3304.4993     4397.6110 -1093.1117
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2291.1852     2743.6285  -452.4433
## TIME_TO_DOM_INTERACTIVE_MS            1797.9580     2411.6625  -613.7044
## TIME_TO_LOAD_EVENT_END_MS             3026.7281     4134.4538 -1107.7257
## TIME_TO_NON_BLANK_PAINT_MS            1457.4383     1837.6274  -380.1891
##                                    eQQ Med  eQQ Mean     eQQ Max
## distance                            0.2500    0.2408      0.3106
## daily_num_sessions_started          0.4000    0.4737      1.5750
## daily_num_sessions_started_max      1.0000    0.8882      3.0000
## FX_PAGE_LOAD_MS_2_PARENT          286.3527  415.2519   1217.0293
## memory_mb                          31.0000  542.1348 196547.0000
## num_active_days                     0.0000    0.2374      1.0000
## num_addons                          2.0000    2.1994    126.6667
## num_bookmarks                       1.0000   79.1581  21769.0000
## profile_age                        23.0000   25.2873   1369.0000
## session_length                      1.4564    2.9903    151.2126
## session_length_max                  2.8661    4.4036    858.3503
## TIME_TO_DOM_COMPLETE_MS           408.7792 1092.1439  11538.8571
## TIME_TO_DOM_CONTENT_LOADED_END_MS 218.3252  451.4889  12439.4915
## TIME_TO_DOM_INTERACTIVE_MS        236.0083  612.6128  20607.0000
## TIME_TO_LOAD_EVENT_END_MS         428.3251 1106.7063  11196.2118
## TIME_TO_NON_BLANK_PAINT_MS        151.5375  379.0529  16102.5273
## 
## 
## Summary of balance by subclasses:
## , , Subclass 1
## 
##                                   Means Treated Means Control   Mean Diff
## distance                                 0.2509        0.2285      0.0225
## daily_num_sessions_started               2.3990        2.0658      0.3332
## daily_num_sessions_started_max           4.4088        3.6718      0.7371
## FX_PAGE_LOAD_MS_2_PARENT              3629.1937     3886.2732   -257.0795
## memory_mb                             9138.1626     8798.5694    339.5932
## num_active_days                          5.5323        5.0662      0.4662
## num_addons                               9.1710        9.2452     -0.0743
## num_bookmarks                          300.3797      295.9452      4.4345
## profile_age                            940.3483      867.1655     73.1827
## session_length                          12.4357       15.0351     -2.5994
## session_length_max                      23.2722       26.7248     -3.4525
## TIME_TO_DOM_COMPLETE_MS               4655.6138     5529.7289   -874.1151
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2943.9156     3218.5432   -274.6276
## TIME_TO_DOM_INTERACTIVE_MS            2547.7871     3027.4856   -479.6985
## TIME_TO_LOAD_EVENT_END_MS             4370.7806     5268.1085   -897.3279
## TIME_TO_NON_BLANK_PAINT_MS            1986.8516     2213.4394   -226.5878
##                                       eQQ Med    eQQ Mean     eQQ Max
## distance                               0.0230      0.0225      0.0412
## daily_num_sessions_started             0.3333      0.3565      5.7500
## daily_num_sessions_started_max         1.0000      0.7739     30.0000
## FX_PAGE_LOAD_MS_2_PARENT             188.9938    280.2886    727.4286
## memory_mb                             40.0000    616.1848  32765.0000
## num_active_days                        0.0000      0.4665      1.0000
## num_addons                             0.3333      0.4939    126.6667
## num_bookmarks                         22.0000     53.0462  22426.0000
## profile_age                           78.0000     74.9695   1231.0000
## session_length                         1.7838      2.6090    151.2126
## session_length_max                     1.7144      3.5516    586.3783
## TIME_TO_DOM_COMPLETE_MS              396.3856    888.4752   6719.6488
## TIME_TO_DOM_CONTENT_LOADED_END_MS    131.1098    280.2402  19981.4000
## TIME_TO_DOM_INTERACTIVE_MS           195.7931    485.9224  22814.3333
## TIME_TO_LOAD_EVENT_END_MS            412.7270    907.7385   6719.6488
## TIME_TO_NON_BLANK_PAINT_MS            65.0801    239.1967  49949.1429
## 
## , , Subclass 2
## 
##                                   Means Treated Means Control   Mean Diff
## distance                                 0.5258        0.5173      0.0085
## daily_num_sessions_started               2.4010        2.3259      0.0751
## daily_num_sessions_started_max           4.3050        4.1744      0.1307
## FX_PAGE_LOAD_MS_2_PARENT              3358.1953     3056.1402    302.0551
## memory_mb                             8490.5660     8053.3879    437.1781
## num_active_days                          5.2675        5.4117     -0.1443
## num_addons                               6.1513        6.3617     -0.2104
## num_bookmarks                          174.3964      125.2153     49.1810
## profile_age                            929.3821      892.7549     36.6271
## session_length                          11.2572        9.8249      1.4323
## session_length_max                      21.6842       18.9055      2.7787
## TIME_TO_DOM_COMPLETE_MS               3838.2941     3220.1870    618.1071
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2504.7020     2131.5823    373.1197
## TIME_TO_DOM_INTERACTIVE_MS            2131.7333     1790.2477    341.4856
## TIME_TO_LOAD_EVENT_END_MS             3572.6793     2980.7217    591.9576
## TIME_TO_NON_BLANK_PAINT_MS            1693.1425     1469.7877    223.3548
##                                       eQQ Med    eQQ Mean     eQQ Max
## distance                               0.0091      0.0085      0.0133
## daily_num_sessions_started             0.1429      0.1641      5.0833
## daily_num_sessions_started_max         0.0000      0.2424      8.0000
## FX_PAGE_LOAD_MS_2_PARENT             213.4000    302.3883    731.4039
## memory_mb                             48.0000    702.4733  65508.0000
## num_active_days                        0.0000      0.1438      1.0000
## num_addons                             0.1667      0.3024      1.3333
## num_bookmarks                         12.0000     55.1384  12400.0000
## profile_age                           36.0000     38.5212   1230.0000
## session_length                         1.0283      1.4521     60.5250
## session_length_max                     2.3419      3.0008    400.1297
## TIME_TO_DOM_COMPLETE_MS              245.0491    618.3153   6224.5733
## TIME_TO_DOM_CONTENT_LOADED_END_MS    190.9594    373.4239   7780.7182
## TIME_TO_DOM_INTERACTIVE_MS           163.5461    341.6326   5029.5595
## TIME_TO_LOAD_EVENT_END_MS            244.5388    592.0556   6676.2247
## TIME_TO_NON_BLANK_PAINT_MS           116.2831    223.1503   4474.0833
## 
## , , Subclass 3
## 
##                                   Means Treated Means Control   Mean Diff
## distance                                 0.6414        0.6312      0.0102
## daily_num_sessions_started               2.5052        3.3211     -0.8159
## daily_num_sessions_started_max           4.4751        6.1976     -1.7225
## FX_PAGE_LOAD_MS_2_PARENT              3082.7227     2697.5427    385.1800
## memory_mb                             8283.3329     9780.1111  -1496.7782
## num_active_days                          5.2337        6.1698     -0.9361
## num_addons                               5.3065        5.9614     -0.6548
## num_bookmarks                          116.6804      180.8574    -64.1770
## profile_age                            868.5729      981.4336   -112.8606
## session_length                           9.6627        6.6615      3.0012
## session_length_max                      18.7577       14.3375      4.4201
## TIME_TO_DOM_COMPLETE_MS               3286.4351     2601.3614    685.0736
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2200.7573     2018.2117    182.5456
## TIME_TO_DOM_INTERACTIVE_MS            1808.2958     1398.2145    410.0814
## TIME_TO_LOAD_EVENT_END_MS             3041.4807     2324.6774    716.8033
## TIME_TO_NON_BLANK_PAINT_MS            1458.5905     1195.1818    263.4087
##                                       eQQ Med    eQQ Mean     eQQ Max
## distance                               0.0118      0.0102      0.0152
## daily_num_sessions_started             0.4286      0.8174      9.0000
## daily_num_sessions_started_max         1.0000      1.7274     16.0000
## FX_PAGE_LOAD_MS_2_PARENT             338.5206    387.0891   1085.0500
## memory_mb                             77.0000   1496.5243  33966.0000
## num_active_days                        1.0000      0.9372      2.0000
## num_addons                             0.8333      0.6633      3.0000
## num_bookmarks                          1.0000     68.2776  21670.0000
## profile_age                          122.0000    114.9576   2620.0000
## session_length                         2.4198      3.0087     15.3549
## session_length_max                     3.4740      4.5009    137.5303
## TIME_TO_DOM_COMPLETE_MS              405.7366    684.6908  14361.0881
## TIME_TO_DOM_CONTENT_LOADED_END_MS    232.4408    378.2134   7292.1079
## TIME_TO_DOM_INTERACTIVE_MS           236.0310    411.0909   9502.8585
## TIME_TO_LOAD_EVENT_END_MS            435.4809    719.0484   5300.0548
## TIME_TO_NON_BLANK_PAINT_MS           170.1881    278.8717   8897.0500
## 
## , , Subclass 4
## 
##                                   Means Treated Means Control   Mean Diff
## distance                                 0.7105        0.7036      0.0070
## daily_num_sessions_started               2.6351        4.9025     -2.2675
## daily_num_sessions_started_max           4.6979        9.2742     -4.5763
## FX_PAGE_LOAD_MS_2_PARENT              2896.2967     2723.3355    172.9612
## memory_mb                             8547.0695    13098.3097  -4551.2402
## num_active_days                          5.2882        6.6333     -1.3451
## num_addons                               4.7944        5.7903     -0.9960
## num_bookmarks                          105.4028      404.5434   -299.1406
## profile_age                            872.5596     1077.3452   -204.7855
## session_length                           8.5216        5.7600      2.7616
## session_length_max                      16.9851       14.1520      2.8331
## TIME_TO_DOM_COMPLETE_MS               2945.9427     2625.1679    320.7748
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2007.8868     2625.5060   -617.6193
## TIME_TO_DOM_INTERACTIVE_MS            1617.0368     1369.8993    247.1375
## TIME_TO_LOAD_EVENT_END_MS             2705.7776     2182.2139    523.5638
## TIME_TO_NON_BLANK_PAINT_MS            1330.7381     1164.8446    165.8935
##                                       eQQ Med    eQQ Mean     eQQ Max
## distance                               0.0075      0.0070      0.0112
## daily_num_sessions_started             1.3750      2.2555     11.2500
## daily_num_sessions_started_max         3.0000      4.5914     30.0000
## FX_PAGE_LOAD_MS_2_PARENT             181.5297    195.7069    853.9716
## memory_mb                           2029.5000   4547.7430  65589.0000
## num_active_days                        2.0000      1.3462      3.0000
## num_addons                             1.0000      1.0063      3.0000
## num_bookmarks                         11.0000    293.8034  20326.3000
## profile_age                          221.0000    203.5376   1370.0000
## session_length                         2.7746      3.1834    140.2416
## session_length_max                     3.4928      6.1577    963.7425
## TIME_TO_DOM_COMPLETE_MS              335.7212    414.9658  11959.6875
## TIME_TO_DOM_CONTENT_LOADED_END_MS    102.6898    741.2994  15199.5967
## TIME_TO_DOM_INTERACTIVE_MS           243.2424    321.6764  15693.5000
## TIME_TO_LOAD_EVENT_END_MS            390.9676    563.0875   4736.1065
## TIME_TO_NON_BLANK_PAINT_MS           153.9446    227.6094  13678.3571
## 
## , , Subclass 5
## 
##                                   Means Treated Means Control   Mean Diff
## distance                                 0.7648        0.7607      0.0041
## daily_num_sessions_started               2.9024        5.7899     -2.8874
## daily_num_sessions_started_max           5.2491       10.7635     -5.5144
## FX_PAGE_LOAD_MS_2_PARENT              2666.1675     2609.6019     56.5656
## memory_mb                             9107.3124    20377.8133 -11270.5009
## num_active_days                          5.6015        6.4191     -0.8176
## num_addons                               4.4209        5.5429     -1.1220
## num_bookmarks                          112.2892      592.6343   -480.3451
## profile_age                            861.0621      980.2988   -119.2367
## session_length                           7.5347        5.1380      2.3968
## session_length_max                      15.2823       11.1722      4.1101
## TIME_TO_DOM_COMPLETE_MS               2625.0234     2780.4329   -155.4096
## TIME_TO_DOM_CONTENT_LOADED_END_MS     1870.5593     3509.5796  -1639.0203
## TIME_TO_DOM_INTERACTIVE_MS            1419.4107     1432.1830    -12.7723
## TIME_TO_LOAD_EVENT_END_MS             2368.0543     2238.5444    129.5099
## TIME_TO_NON_BLANK_PAINT_MS            1189.4232     1187.2014      2.2218
##                                       eQQ Med    eQQ Mean     eQQ Max
## distance                               0.0045      0.0041      0.0072
## daily_num_sessions_started             1.3000      2.8410     13.7083
## daily_num_sessions_started_max         3.0000      5.4315     39.0000
## FX_PAGE_LOAD_MS_2_PARENT              95.0470    121.8725    826.9945
## memory_mb                           8183.0000  11399.3195  57277.0000
## num_active_days                        1.0000      0.8340      2.0000
## num_addons                             1.0000      1.1622      2.0000
## num_bookmarks                         14.0000    430.0617  18798.0000
## profile_age                          111.0000    112.0373    856.0000
## session_length                         2.2544      2.5946     25.4690
## session_length_max                     3.4842      5.3169    211.4811
## TIME_TO_DOM_COMPLETE_MS              187.9138    441.8738  31221.1000
## TIME_TO_DOM_CONTENT_LOADED_END_MS     66.5836   1637.8773  17465.4796
## TIME_TO_DOM_INTERACTIVE_MS           127.3965    224.3370   6019.0057
## TIME_TO_LOAD_EVENT_END_MS            279.2186    341.0763   5009.2917
## TIME_TO_NON_BLANK_PAINT_MS           112.7420    393.1289  49277.3333
## 
## , , Subclass 6
## 
##                                   Means Treated Means Control   Mean Diff
## distance                                 0.8272        0.8505     -0.0233
## daily_num_sessions_started               4.3318        4.9130     -0.5812
## daily_num_sessions_started_max           8.0604        9.6513     -1.5909
## FX_PAGE_LOAD_MS_2_PARENT              2632.2594     2751.3244   -119.0650
## memory_mb                            12686.3512    30182.8421 -17496.4909
## num_active_days                          6.4459        5.7697      0.6761
## num_addons                               4.1663        4.5669     -0.4006
## num_bookmarks                          179.2124     2237.2700  -2058.0576
## profile_age                            923.5093      896.8355     26.6738
## session_length                           5.8352        9.0870     -3.2518
## session_length_max                      12.8894       28.3594    -15.4700
## TIME_TO_DOM_COMPLETE_MS               2475.6838     2774.1794   -298.4956
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2219.2748     3771.2138  -1551.9390
## TIME_TO_DOM_INTERACTIVE_MS            1263.4862     1659.4946   -396.0083
## TIME_TO_LOAD_EVENT_END_MS             2101.5986     2478.4162   -376.8176
## TIME_TO_NON_BLANK_PAINT_MS            1085.8839     1442.3090   -356.4251
##                                       eQQ Med    eQQ Mean     eQQ Max
## distance                               0.0169      0.0231      0.0810
## daily_num_sessions_started             0.4375      0.9971      7.5714
## daily_num_sessions_started_max         1.0000      2.5724     29.0000
## FX_PAGE_LOAD_MS_2_PARENT              99.6082    187.1169   1715.6173
## memory_mb                           1405.5000  18303.7961 196547.0000
## num_active_days                        1.0000      0.6711      2.0000
## num_addons                             1.6333      1.6563      4.0000
## num_bookmarks                          5.0000   1982.8786  31046.4286
## profile_age                           34.0000     83.9211   2559.0000
## session_length                         0.4472      2.9981    121.9651
## session_length_max                     0.7782     13.6103   1033.4492
## TIME_TO_DOM_COMPLETE_MS               95.0598    458.3183  13187.8061
## TIME_TO_DOM_CONTENT_LOADED_END_MS    123.3698   1429.7080  11419.9288
## TIME_TO_DOM_INTERACTIVE_MS            50.9078    360.5745   7145.0000
## TIME_TO_LOAD_EVENT_END_MS             52.9427    404.1767   4622.9001
## TIME_TO_NON_BLANK_PAINT_MS            43.7286    741.9514  74840.5000
## 
## 
## Sample sizes by subclasses:
##         Subclass 1 Subclass 2 Subclass 3 Subclass 4 Subclass 5 Subclass 6
## Treated       5829       5829       5828       5829       5829       5829
## Control      19144      10932       3628        930        241        152
## Total        24973      16761       9456       6759       6070       5981
## 
## Summary of balance across subclasses
##                                   Means Treated Means Control Mean Diff
## distance                                 0.6201        0.6153    0.0060
## daily_num_sessions_started               2.8624        3.8864    0.6368
## daily_num_sessions_started_max           5.1994        7.2888    1.2628
## FX_PAGE_LOAD_MS_2_PARENT              3044.1381     2954.0437   99.0162
## memory_mb                             9375.4970    15048.6562 3560.7322
## num_active_days                          5.5615        5.9116    0.3354
## num_addons                               5.6684        6.2448    0.2833
## num_bookmarks                          164.7282      639.4241  356.0054
## profile_age                            899.2399      949.3047   46.0377
## session_length                           9.2079        8.5845    1.0770
## session_length_max                      18.1451       18.9420    2.9032
## TIME_TO_DOM_COMPLETE_MS               3304.4993     3255.1950  225.5580
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2291.1852     2879.1307  398.7701
## TIME_TO_DOM_INTERACTIVE_MS            1797.9580     1779.5984  142.6862
## TIME_TO_LOAD_EVENT_END_MS             3026.7281     2912.1305  241.6520
## TIME_TO_NON_BLANK_PAINT_MS            1457.4383     1445.4678   95.0399
##                                     eQQ Med  eQQ Mean    eQQ Max
## distance                             0.0121    0.0126     0.0282
## daily_num_sessions_started           0.6696    1.2386     8.7272
## daily_num_sessions_started_max       1.5000    2.5565    25.3336
## FX_PAGE_LOAD_MS_2_PARENT           186.1789  245.7397   990.0750
## memory_mb                         1963.8873 6177.8073 75276.5145
## num_active_days                      0.8333    0.7331     1.8333
## num_addons                           0.8278    0.8807    23.3339
## num_bookmarks                       10.8336  480.5461 21111.1054
## profile_age                        100.3327  104.6571  1644.3054
## session_length                       1.7847    2.6410    85.7967
## session_length_max                   2.5476    6.0231   555.4638
## TIME_TO_DOM_COMPLETE_MS            277.6407  584.4370 13945.6388
## TIME_TO_DOM_CONTENT_LOADED_END_MS  141.1896  806.8060 13190.0405
## TIME_TO_DOM_INTERACTIVE_MS         169.4842  357.5374 11034.0866
## TIME_TO_LOAD_EVENT_END_MS          302.6421  587.8601  5510.7104
## TIME_TO_NON_BLANK_PAINT_MS         110.3260  350.6535 33520.1152
## 
## Percent Balance Improvement:
##                                   Mean Diff.    eQQ Med   eQQ Mean   eQQ Max
## distance                             97.9984    95.1456    94.7853   90.9244
## daily_num_sessions_started         -116.8928   -67.3876  -161.4850 -454.1063
## daily_num_sessions_started_max     -135.9096   -50.0014  -187.8414 -744.4533
## FX_PAGE_LOAD_MS_2_PARENT             78.3081    34.9827    40.8215   18.6482
## memory_mb                         -1246.8861 -6235.1203 -1039.5335   61.7005
## num_active_days                     -59.7164       -Inf  -208.8670  -83.3329
## num_addons                           73.7930    58.6111    59.9566   81.5785
## num_bookmarks                      -498.1511  -983.3615  -507.0711    3.0222
## profile_age                        -768.4716  -336.2292  -313.8720  -20.1100
## session_length                       79.1762   -22.5368    11.6827   43.2609
## session_length_max                   81.9806    11.1142   -36.7763   35.2870
## TIME_TO_DOM_COMPLETE_MS              95.4895    32.0805    46.4872  -20.8581
## TIME_TO_DOM_CONTENT_LOADED_END_MS   -29.9490    35.3306   -78.6990   -6.0336
## TIME_TO_DOM_INTERACTIVE_MS           97.0084    28.1872    41.6373   46.4547
## TIME_TO_LOAD_EVENT_END_MS            89.6547    29.3429    46.8820   50.7806
## TIME_TO_NON_BLANK_PAINT_MS           96.8515    27.1956     7.4922 -108.1668

Match using Subclassification matching: Full Dataset

## 
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x, 
##     method = "subclass", sub.by = "treat")
## Summary of balance for all data:
##                                   Means Treated Means Control  Mean Diff
## distance                                 0.6213        0.3787     0.2426
## daily_num_sessions_started               2.8876        2.3689     0.5188
## daily_num_sessions_started_max           5.2501        4.2814     0.9687
## FX_PAGE_LOAD_MS_2_PARENT              3027.5228     3463.7084  -436.1856
## memory_mb                             9460.0891     8965.1561   494.9330
## num_active_days                          5.5746        5.3462     0.2284
## num_addons                               5.6632        7.8554    -2.1922
## num_bookmarks                          164.1029      242.4878   -78.3849
## profile_age                            893.9339      893.7534     0.1805
## session_length                           9.2495       12.2962    -3.0467
## session_length_max                      18.1985       22.7066    -4.5080
## TIME_TO_DOM_COMPLETE_MS               3285.4718     4388.9143 -1103.4424
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2292.5533     2737.6381  -445.0848
## TIME_TO_DOM_INTERACTIVE_MS            1792.4896     2404.3932  -611.9036
## TIME_TO_LOAD_EVENT_END_MS             3012.8713     4126.9123 -1114.0410
## TIME_TO_NON_BLANK_PAINT_MS            1449.9423     1833.6557  -383.7134
##                                    eQQ Med  eQQ Mean     eQQ Max
## distance                            0.2527    0.2426      0.3114
## daily_num_sessions_started          0.4167    0.5190      1.7500
## daily_num_sessions_started_max      1.0000    0.9692     11.0000
## FX_PAGE_LOAD_MS_2_PARENT          299.6662  436.2548   1250.9746
## memory_mb                          31.0000  555.3761 262176.0000
## num_active_days                     0.0000    0.2413      1.0000
## num_addons                          2.0000    2.1941    109.0000
## num_bookmarks                       1.0000   79.0181  21769.0000
## profile_age                        22.0000   23.1161   1366.0000
## session_length                      1.4538    3.0489    151.4597
## session_length_max                  2.7333    4.5187    871.0939
## TIME_TO_DOM_COMPLETE_MS           421.7080 1103.4946  12182.3743
## TIME_TO_DOM_CONTENT_LOADED_END_MS 229.0178  445.1832  10634.8539
## TIME_TO_DOM_INTERACTIVE_MS        243.7872  612.0582  19062.0000
## TIME_TO_LOAD_EVENT_END_MS         441.3178 1114.0691  11196.2118
## TIME_TO_NON_BLANK_PAINT_MS        155.3258  384.1746  19260.1667
## 
## 
## Summary of balance by subclasses:
## , , Subclass 1
## 
##                                   Means Treated Means Control   Mean Diff
## distance                                 0.2516        0.2302      0.0214
## daily_num_sessions_started               2.4101        2.0198      0.3903
## daily_num_sessions_started_max           4.4171        3.6037      0.8134
## FX_PAGE_LOAD_MS_2_PARENT              3645.6022     3894.3402   -248.7380
## memory_mb                             9111.6550     8806.4754    305.1795
## num_active_days                          5.5209        5.0730      0.4480
## num_addons                               9.1570        9.1982     -0.0412
## num_bookmarks                          305.6801      287.6036     18.0765
## profile_age                            942.1902      872.1816     70.0086
## session_length                          12.3416       15.1584     -2.8168
## session_length_max                      23.1106       26.8892     -3.7786
## TIME_TO_DOM_COMPLETE_MS               4672.6893     5493.3005   -820.6112
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2969.6198     3189.9871   -220.3673
## TIME_TO_DOM_INTERACTIVE_MS            2563.9265     2997.3077   -433.3811
## TIME_TO_LOAD_EVENT_END_MS             4383.9143     5234.3483   -850.4340
## TIME_TO_NON_BLANK_PAINT_MS            2013.0789     2193.1474   -180.0685
##                                       eQQ Med    eQQ Mean     eQQ Max
## distance                               0.0223      0.0214      0.0368
## daily_num_sessions_started             0.3393      0.3917      2.6250
## daily_num_sessions_started_max         1.0000      0.8153     11.0000
## FX_PAGE_LOAD_MS_2_PARENT             164.4150    273.4149    691.8043
## memory_mb                             34.0000    558.1491  65529.0000
## num_active_days                        0.0000      0.4480      1.0000
## num_addons                             0.3333      0.4836    109.0000
## num_bookmarks                         22.0000     56.0556  22426.0000
## profile_age                           72.0000     71.1095   1228.0000
## session_length                         1.8137      2.8224    151.2126
## session_length_max                     1.5853      3.8216    586.3783
## TIME_TO_DOM_COMPLETE_MS              348.9241    834.4865   3311.9972
## TIME_TO_DOM_CONTENT_LOADED_END_MS     93.7953    244.5282  15414.5714
## TIME_TO_DOM_INTERACTIVE_MS           167.5294    446.8677  18812.3581
## TIME_TO_LOAD_EVENT_END_MS            369.7923    860.9821   3421.6268
## TIME_TO_NON_BLANK_PAINT_MS            47.9215    197.2894  41949.0000
## 
## , , Subclass 2
## 
##                                   Means Treated Means Control   Mean Diff
## distance                                 0.5271        0.5179      0.0092
## daily_num_sessions_started               2.3838        2.3032      0.0807
## daily_num_sessions_started_max           4.2884        4.1319      0.1565
## FX_PAGE_LOAD_MS_2_PARENT              3354.8414     3054.6816    300.1598
## memory_mb                             8518.6848     8010.2872    508.3976
## num_active_days                          5.2812        5.4200     -0.1387
## num_addons                               6.1290        6.3419     -0.2129
## num_bookmarks                          168.1025      136.4346     31.6679
## profile_age                            924.5122      898.0909     26.4212
## session_length                          11.2709        9.8223      1.4486
## session_length_max                      21.6102       19.0209      2.5893
## TIME_TO_DOM_COMPLETE_MS               3822.8501     3219.8669    602.9832
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2476.8607     2140.6634    336.1973
## TIME_TO_DOM_INTERACTIVE_MS            2111.5677     1793.6102    317.9575
## TIME_TO_LOAD_EVENT_END_MS             3556.3707     2979.4544    576.9163
## TIME_TO_NON_BLANK_PAINT_MS            1673.8577     1471.3043    202.5534
##                                       eQQ Med    eQQ Mean     eQQ Max
## distance                               0.0100      0.0092      0.0139
## daily_num_sessions_started             0.1250      0.1477      4.3750
## daily_num_sessions_started_max         0.0000      0.2282     10.0000
## FX_PAGE_LOAD_MS_2_PARENT             222.0587    300.3449    722.5508
## memory_mb                             42.0000    621.9851  65458.0000
## num_active_days                        0.0000      0.1386      1.0000
## num_addons                             0.1667      0.3129      1.5000
## num_bookmarks                         12.1429     47.0343  14227.0000
## profile_age                           24.0000     27.3316    492.0000
## session_length                         1.0825      1.4915    100.8215
## session_length_max                     2.2539      3.0444    992.3211
## TIME_TO_DOM_COMPLETE_MS              243.5112    603.2329   9796.1429
## TIME_TO_DOM_CONTENT_LOADED_END_MS    169.5215    337.2614   8263.6343
## TIME_TO_DOM_INTERACTIVE_MS           146.1880    318.0017   8589.9028
## TIME_TO_LOAD_EVENT_END_MS            232.4506    576.8566   6005.7348
## TIME_TO_NON_BLANK_PAINT_MS           110.2811    202.7836   4541.0000
## 
## , , Subclass 3
## 
##                                   Means Treated Means Control   Mean Diff
## distance                                 0.6420        0.6325      0.0094
## daily_num_sessions_started               2.4619        3.3168     -0.8549
## daily_num_sessions_started_max           4.4122        6.1918     -1.7796
## FX_PAGE_LOAD_MS_2_PARENT              3086.4699     2659.3165    427.1534
## memory_mb                             8344.9531     9961.1075  -1616.1544
## num_active_days                          5.2810        6.1962     -0.9152
## num_addons                               5.2979        5.9753     -0.6774
## num_bookmarks                          121.6145      191.1804    -69.5658
## profile_age                            876.7948      971.0145    -94.2197
## session_length                           9.8206        6.6121      3.2085
## session_length_max                      19.0226       14.4456      4.5770
## TIME_TO_DOM_COMPLETE_MS               3284.0538     2580.3769    703.6769
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2193.2978     2004.0599    189.2378
## TIME_TO_DOM_INTERACTIVE_MS            1807.3320     1400.5442    406.7879
## TIME_TO_LOAD_EVENT_END_MS             3043.2840     2291.9163    751.3678
## TIME_TO_NON_BLANK_PAINT_MS            1460.3143     1202.5560    257.7584
##                                       eQQ Med    eQQ Mean     eQQ Max
## distance                               0.0108      0.0094      0.0140
## daily_num_sessions_started             0.5250      0.8557      7.5500
## daily_num_sessions_started_max         1.0000      1.7844     29.0000
## FX_PAGE_LOAD_MS_2_PARENT             366.1511    428.3443   1136.7030
## memory_mb                             75.0000   1617.7099  32761.0000
## num_active_days                        1.0000      0.9157      2.0000
## num_addons                             0.9167      0.6798      2.4545
## num_bookmarks                          1.0000     75.2084  19765.0000
## profile_age                          102.0000     98.8166   2052.0000
## session_length                         2.7476      3.2756    123.8998
## session_length_max                     4.2128      5.2575    995.9783
## TIME_TO_DOM_COMPLETE_MS              440.2364    704.7485   3050.4283
## TIME_TO_DOM_CONTENT_LOADED_END_MS    245.2150    371.2476   5581.5286
## TIME_TO_DOM_INTERACTIVE_MS           253.5394    411.1351   3351.0788
## TIME_TO_LOAD_EVENT_END_MS            465.0222    752.1617   3870.4200
## TIME_TO_NON_BLANK_PAINT_MS           178.8468    300.0734  37901.1429
## 
## , , Subclass 4
## 
##                                   Means Treated Means Control   Mean Diff
## distance                                 0.7114        0.7050      0.0064
## daily_num_sessions_started               2.6317        5.0880     -2.4563
## daily_num_sessions_started_max           4.7066        9.6201     -4.9135
## FX_PAGE_LOAD_MS_2_PARENT              2899.4335     2666.3722    233.0612
## memory_mb                             8514.6531    13264.7341  -4750.0809
## num_active_days                          5.3143        6.5885     -1.2742
## num_addons                               4.7867        5.7945     -1.0078
## num_bookmarks                          103.0137      283.8290   -180.8153
## profile_age                            864.8687      999.2260   -134.3573
## session_length                           8.6304        5.5841      3.0463
## session_length_max                      17.3249       12.8122      4.5126
## TIME_TO_DOM_COMPLETE_MS               2945.8132     2649.6120    296.2013
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2025.2565     2654.9082   -629.6517
## TIME_TO_DOM_INTERACTIVE_MS            1617.3892     1370.7463    246.6429
## TIME_TO_LOAD_EVENT_END_MS             2699.1145     2191.5727    507.5417
## TIME_TO_NON_BLANK_PAINT_MS            1324.0560     1157.6885    166.3676
##                                       eQQ Med    eQQ Mean     eQQ Max
## distance                               0.0068      0.0064      0.0103
## daily_num_sessions_started             1.7143      2.4516     11.2500
## daily_num_sessions_started_max         4.0000      4.9041     30.0000
## FX_PAGE_LOAD_MS_2_PARENT             239.0448    246.9080    764.8333
## memory_mb                           2248.0000   4722.5969  33966.0000
## num_active_days                        1.0000      1.2750      2.0000
## num_addons                             1.0000      1.0189      3.2000
## num_bookmarks                         11.0000    175.9415  14549.0000
## profile_age                          149.0000    135.4804   1604.0000
## session_length                         2.8483      3.0998     15.5721
## session_length_max                     3.5917      5.2588    209.0297
## TIME_TO_DOM_COMPLETE_MS              325.4478    449.0558  23673.0966
## TIME_TO_DOM_CONTENT_LOADED_END_MS    112.6439    777.1600  11550.4087
## TIME_TO_DOM_INTERACTIVE_MS           246.9167    331.1044  13564.9171
## TIME_TO_LOAD_EVENT_END_MS            391.9440    552.3713   8816.9173
## TIME_TO_NON_BLANK_PAINT_MS           168.9755    242.1909  10601.6000
## 
## , , Subclass 5
## 
##                                   Means Treated Means Control   Mean Diff
## distance                                 0.7657        0.7617      0.0040
## daily_num_sessions_started               2.8739        6.3624     -3.4885
## daily_num_sessions_started_max           5.2190       11.7263     -6.5073
## FX_PAGE_LOAD_MS_2_PARENT              2633.8482     2562.6295     71.2187
## memory_mb                             9125.3363    19293.1744 -10167.8381
## num_active_days                          5.6194        6.5342     -0.9148
## num_addons                               4.4228        5.6194     -1.1966
## num_bookmarks                          109.5189      453.3226   -343.8037
## profile_age                            855.5199      940.0817    -84.5618
## session_length                           7.6116        5.1372      2.4744
## session_length_max                      15.2832       11.1064      4.1768
## TIME_TO_DOM_COMPLETE_MS               2583.9345     2466.1432    117.7913
## TIME_TO_DOM_CONTENT_LOADED_END_MS     1878.4901     3370.1628  -1491.6727
## TIME_TO_DOM_INTERACTIVE_MS            1409.5434     1378.0845     31.4589
## TIME_TO_LOAD_EVENT_END_MS             2331.8125     2111.0402    220.7723
## TIME_TO_NON_BLANK_PAINT_MS            1174.0838     1256.6035    -82.5197
##                                       eQQ Med    eQQ Mean     eQQ Max
## distance                               0.0043      0.0040      0.0062
## daily_num_sessions_started             1.5833      3.4714     13.7083
## daily_num_sessions_started_max         3.0000      6.4901     23.0000
## FX_PAGE_LOAD_MS_2_PARENT             121.7924    164.3502   1622.7155
## memory_mb                           7495.0000   9993.7042  49113.0000
## num_active_days                        1.0000      0.9249      2.0000
## num_addons                             1.0000      1.2433      2.6667
## num_bookmarks                         11.7500    317.9802  18798.0000
## profile_age                           87.0000     83.0508   1351.0000
## session_length                         2.1457      2.5583     25.4151
## session_length_max                     2.8511      4.8221    169.2850
## TIME_TO_DOM_COMPLETE_MS              257.8807    354.0923  10297.8289
## TIME_TO_DOM_CONTENT_LOADED_END_MS     85.9501   1537.6741  14560.9670
## TIME_TO_DOM_INTERACTIVE_MS           157.4279    239.3562   6359.4583
## TIME_TO_LOAD_EVENT_END_MS            300.3357    379.8884   4270.1067
## TIME_TO_NON_BLANK_PAINT_MS           118.9085    266.0568  28137.7476
## 
## , , Subclass 6
## 
##                                   Means Treated Means Control   Mean Diff
## distance                                 0.8300        0.8482     -0.0182
## daily_num_sessions_started               4.5646        6.0655     -1.5010
## daily_num_sessions_started_max           8.4573       11.5828     -3.1254
## FX_PAGE_LOAD_MS_2_PARENT              2544.9476     2675.1385   -130.1909
## memory_mb                            13145.1402    27851.7517 -14706.6115
## num_active_days                          6.4306        5.9139      0.5167
## num_addons                               4.1856        4.7301     -0.5444
## num_bookmarks                          176.6831     2294.1172  -2117.4340
## profile_age                            899.7160      866.1954     33.5207
## session_length                           5.8220        7.6336     -1.8116
## session_length_max                      12.8399       21.4820     -8.6421
## TIME_TO_DOM_COMPLETE_MS               2403.4901     2629.9196   -226.4295
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2211.7849     3753.0673  -1541.2824
## TIME_TO_DOM_INTERACTIVE_MS            1245.1803     1571.1946   -326.0142
## TIME_TO_LOAD_EVENT_END_MS             2062.7350     2373.7297   -310.9947
## TIME_TO_NON_BLANK_PAINT_MS            1054.2640     1461.4806   -407.2166
##                                       eQQ Med    eQQ Mean     eQQ Max
## distance                               0.0068      0.0184      0.0751
## daily_num_sessions_started             0.3429      1.8072     11.0000
## daily_num_sessions_started_max         1.0000      3.7781     25.0000
## FX_PAGE_LOAD_MS_2_PARENT              70.3052    183.7392   1999.6909
## memory_mb                           1938.0000  15460.0099 262176.0000
## num_active_days                        0.0000      0.5132      2.0000
## num_addons                             1.6667      1.6546      4.0000
## num_bookmarks                          2.0000   2085.3162  32271.0417
## profile_age                           39.0000     77.8344   2274.0000
## session_length                         0.1678      1.8351    119.4322
## session_length_max                     0.3888      8.3654   1021.1394
## TIME_TO_DOM_COMPLETE_MS              103.8242    374.9957  13187.8061
## TIME_TO_DOM_CONTENT_LOADED_END_MS    156.2572   1501.2906  12944.3276
## TIME_TO_DOM_INTERACTIVE_MS            49.7759    308.9210  13466.7365
## TIME_TO_LOAD_EVENT_END_MS             91.5503    390.9446   4908.1131
## TIME_TO_NON_BLANK_PAINT_MS            38.8486    518.6575  59385.5000
## 
## 
## Sample sizes by subclasses:
##         Subclass 1 Subclass 2 Subclass 3 Subclass 4 Subclass 5 Subclass 6
## Treated       9938       9938       9937       9938       9938       9938
## Control      32931      18517       5871       1553        453        302
## Total        42869      28455      15808      11491      10391      10240
## 
## Summary of balance across subclasses
##                                   Means Treated Means Control Mean Diff
## distance                                 0.6213        0.6159    0.0053
## daily_num_sessions_started               2.8876        4.1926    0.7700
## daily_num_sessions_started_max           5.2501        7.8095    1.4917
## FX_PAGE_LOAD_MS_2_PARENT              3027.5228     2918.7508  106.8154
## memory_mb                             9460.0891    14531.3317 3096.6255
## num_active_days                          5.5746        5.9543    0.3242
## num_addons                               5.6632        6.2766    0.3005
## num_bookmarks                          164.1029      607.7549  359.0398
## profile_age                            893.9339      924.4642   33.6660
## session_length                           9.2495        8.3247    1.0410
## session_length_max                      18.1985       17.6261    2.0711
## TIME_TO_DOM_COMPLETE_MS               3285.4718     3173.2131  216.3453
## TIME_TO_DOM_CONTENT_LOADED_END_MS     2292.5533     2852.1557  379.8636
## TIME_TO_DOM_INTERACTIVE_MS            1792.4896     1751.9205  131.4965
## TIME_TO_LOAD_EVENT_END_MS             3012.8713     2863.6865  237.0909
## TIME_TO_NON_BLANK_PAINT_MS            1449.9423     1457.1343   97.2115
##                                     eQQ Med  eQQ Mean    eQQ Max
## distance                             0.0102    0.0115     0.0260
## daily_num_sessions_started           0.7716    1.5209     8.4181
## daily_num_sessions_started_max       1.6667    3.0000    21.3332
## FX_PAGE_LOAD_MS_2_PARENT           197.2917  266.1809  1156.3833
## memory_mb                         1972.0318 5495.7576 84834.7066
## num_active_days                      0.5000    0.7026     1.6667
## num_addons                           0.8472    0.8988    20.4705
## num_bookmarks                        9.9823  459.5958 20339.3499
## profile_age                         78.8329   82.2702  1500.1574
## session_length                       1.8009    2.5138    89.3916
## session_length_max                   2.4805    5.0950   662.3497
## TIME_TO_DOM_COMPLETE_MS            286.6349  553.4327 10553.0092
## TIME_TO_DOM_CONTENT_LOADED_END_MS  143.8955  794.8674 11386.0036
## TIME_TO_DOM_INTERACTIVE_MS         170.2282  342.5632 10690.8650
## TIME_TO_LOAD_EVENT_END_MS          308.5132  585.5313  5215.5090
## TIME_TO_NON_BLANK_PAINT_MS         110.6292  287.8417 30419.2063
## 
## Percent Balance Improvement:
##                                    Mean Diff.    eQQ Med  eQQ Mean   eQQ Max
## distance                              97.7885    95.9832   95.2700   91.6337
## daily_num_sessions_started          -151.5586   -85.1915 -193.0205 -381.0326
## daily_num_sessions_started_max      -164.2034   -66.6678 -209.5455  -93.9382
## FX_PAGE_LOAD_MS_2_PARENT              75.0629    34.1628   38.9850    7.5614
## memory_mb                           -924.6321 -6261.3929 -889.5560   67.6421
## num_active_days                      -66.2352       -Inf -191.1955  -66.6661
## num_addons                            72.0196    57.6389   59.0349   81.2197
## num_bookmarks                       -465.9914  -898.2293 -481.6335    6.5674
## profile_age                       -16810.6400  -258.3316 -255.9004   -9.8212
## session_length                        69.6434   -23.8817   17.5515   40.9799
## session_length_max                    87.3018     9.2482  -12.7543   23.9635
## TIME_TO_DOM_COMPLETE_MS               89.8265    32.0300   49.8473   13.3748
## TIME_TO_DOM_CONTENT_LOADED_END_MS    -25.7294    37.1684  -78.5484   -7.0631
## TIME_TO_DOM_INTERACTIVE_MS            93.3700    30.1735   44.0309   43.9153
## TIME_TO_LOAD_EVENT_END_MS             86.6087    30.0927   47.4421   53.4172
## TIME_TO_NON_BLANK_PAINT_MS            98.1257    28.7760   25.0753  -57.9384

Graphical Comparison

  • Note that we want to see if the Matched Treated (beta) and Matched Control (release) distributions are roughly similar.

Full Dataset

## [1] "cem_full_models_1x"

## [1] "nn_full_models_1x"

## [1] "nn.ma_full_models_1x"
## [1] "Not appropriate for pure Mahalanobis matching. No plots generated."
## [1] "sub_full_models_1x"


Observations

  • Although it was not possible to graphically compare the results using the Nearest Neighbor matching with Mahalanobis distance, from the summary, we can see that those models outperformed the others. Analyzing each of the covariates, we have:

Full Dataset

  • The figures show that the model containing the entire dataset was better than sampling.
  • In addition, the main problem is displayed mainly in pageload metrics.